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Abstract 

We propose a model for evolutionary game dynamics with three strategies A, B 
and C in the framework of Moran process in finite populations. The model can be 
described as a stochastic process which can be numerically computed from a system 
of linear equations. Furthermore, to capture the feature of the evolutionary process, 
we define two essential variables, the global and the local fixation probability. If the 
global fixation probability of strategy A exceeds the neutral fixation probability, 
the selection favors A replacing B or C no matter what the initial ratio of B to 
C is. Similarly, if the local fixation probability of A exceeds the neutral one, the 
selection favors A replacing B or C only in some appropriate initial ratios of B to 
C. Besides, using our model, the famous game with AllC, AllD and TFT is analyzed. 
Meanwhile, we find that a single individual TFT could invade the entire population 
under proper conditions. 

Key words: evolutionary game theory, three strategies, finite populations, global 
fixation probability, local fixation probability 
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1 Introduction 



Since tlie theory of games was first explicitly applied in evolutionary biology 
by Lewontin [1], it has undergone extensive development. Evolutionary game 
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theory as a mathematical framework to depict the popTilation dynamics under 
natural selection has attracted much attention for a long time. It is a way of 
thinking about particular phenotypes depending on their frequencies in the 
population [2]. Much more important results in terms of the corresponding 
replicator dynamics [3,4], in which the size of the well- mixed population is in- 
finite, promote the development of the evolutionary game theory. Replicator 
dynamics, which is due to Taylor and Jonker [5], is a system of determinis- 
tic differential equations which could describe the evolutionary dynamics of 
multi-species. Besides replicator dynamics, the Lotka-Volterra equations which 
were devised by Lotka and Volterra, have received much attention. They are 
the most common models for the dynamics of the population numbers [3,6], 
whereas replicator dynamics is the most common model for the evolution of 
the frequencies of strategies in a population. These two deterministic models 
both fail to account for stochastic effects. Thus the theory of stochastic pro- 
cesses plays an extraordinarily important role in depicting the evolutionary 
dynamics. 

In nature, however, populations are finite in size. Finite population effects 
can be neglected in infinite populations, but affect the evolution in finite size. 
In fact, with high probability, the state of the process for large size N re- 
mains close to the results of corresponding deterministic replicator dynamics 
for some large time T [7]. Recently, an explicit mean- field description in the 
form of Fokker-Planck equation was derived for frequency-dependent selection 
in finite populations [8,9,10]. It is an approach which could connect the situa- 
tion of finite populations with that of infinite populations. The explicit rules 
governing the interaction of a finite number of individuals with each other 
are embodied in a master equation. And the finite size effects are captured in 
the drift and diffusion terms of a Fokker-Planck equation where the diffusion 
term vanishes with for increasing population sizes. This framework was 
extended to an evolutionary game with an arbitrary number of strategies [11]. 
The stochastic evolutionary processes would be characterized by the stochas- 
tic replicator-mutator equation Xi = ai{x) + Y.'jZl Cij{x)^j{t), here Xi means 
the density of the ith individuals using one of the arbitrary d strategies. Note 
that the difference between above equation and the replicator dynamics in 
infinite populations is the uncorrelated Gaussian noises term Zlj=i Cij{x)^j{t). 
Thus the finite size effects can be viewed as combinations of some uncorrelated 
Gaussian noises. 

Nowak introduced the frequency-dependent Moran process into evolutionary 
game in finite populations [12,13,14]. The frequency-dependent Moran process 
is a stochastic birth-death process. It follows two steps: selection, a player is 
selected to reproduce with a probability proportional to its fitness, and the 
offspring will use the same strategy as its parent; replacement, a randomly 
selected individual is replaced by the offspring. Hence, the population size, 
A^, is strictly constant [15]. Suppose a population consists of individuals who 
use either strategy A oi B. Individual using strategy A receives payoff a or 
b playing with A or B individual; individual using strategy B obtains payoff 
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c or 0? playing with A ot B individual. Viewing individuals using strategy A 
as mutants, we get the probability x{i) that i mutants could invade and take 
over the whole populations. The fitness of individuals using strategy A and B 
is respectively given by: 

fi^l-w + w[a{i - 1) + biN - i)]/{N - 1) 
gi^l-w + w[ci + d{N-i- 1)]/{N - 1) 



Here i = 1,2, ■ ■ ■ , N — 1, w & [0, 1] describes the contribution of the game to 
the fitness. For neutral selection, it needs w — 0; for weak selection, it must 
satisfy the condition ty ^ 1. Accordingly, the fixation probability x{i) is given 
by [16]: 

x{Q) = Q,x{N) = 1 

1+Enf 



In the limit of weak selection, the 1/3 law can be obtained. If A and B are 
strict Nash equilibria and the unstable equilibrium occurs at a frequency of 
A which is less than 1/3, then selection favors replacement oi B hy A. More- 
over, stochastic evolution of finite populations need not choose the strict Nash 
equilibrium and can therefore favor cooperation over defection [17]. 
Obviously, the characteristic timescales also play a crucial role in the evolu- 
tionary dynamics. Sometimes, although the mutant could invade the entire 
population, it takes such a long time that the population typically consists 
of coexisting strategies [18]. It can be shown that a single mutant following 
strategy A fixates in the same average time as a single B individual does in 
a given game, although the fixation probability for the two strategies are dif- 
ferent [19]. Furthermore, if the population size is appropriate for the fixation 
of the cooperative strategy, then this fixation will be fast [20]. Besides the 
standard Moran process, Wright-Fisher model and pairwise comparison using 
Fermi function are brought into the analysis of the evolutionary game in finite 
populations [21,22]. Moreover, spatial structure effects can not be ignored in 
the real world. Much results reveal that a proper spatial structure could en- 
hance the fixation probabihty [23,24]. 

Most results state the situation with two strategies. But in reality, there may 

be many strategies in a game. Furthermore, in coevolution of three strategies, 
how and why a single A individual could invade a finite population of B and 
C individuals, what kinds of strategists would be washed out by the natural 
selection, and how cooperation could emerge in finite populations are unclear. 
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Motivated by these, here we study the evolutionary game of finite populations 
with three strategies. This paper is organized as follows. Using the stochas- 
tic processes theory, we formulate the evolutionary game dynamics in finite 
populations as a system of hnear equations in Section 2. The variable of these 
equations is fixation probability x{i,j), which represents the probability that 
i individuals using strategy A could dominate a population in which j of them 
follow strategy B and N — i—j follow strategy C. Two probabilities, the global 
and the local fixation probabihty, act crucial roles in the evolutionary game 
dynamics with three strategies. If the global fixation probability of a single A 
individual exceeds the neutral fixation probability 1/N, the selection favors A 
replacing or C no matter what the initial ratio of S to C is. Similarly, if the 
/oca/ fixation probability exceeds the neutral one the selection favors A 

replacing B or C only in some appropriate initial ratios of B to C. In Section 
3, some numeric computations of evolutionary game with AUG, AUD and TFT 
are adopted to investigate the emergence of cooperation in some specified sit- 
uations. For weak selection and sufficiently large size N, wc find a condition 
in terms of the number of rounds n and the ratio r of cost to benefit, under 
which the selection favors only one TFT replacing AUG or AUD individuals. 
Furthermore, the condition under which a single TFT could invade the entire 
population is also obtained. Finally, the results are summarized and discussed 
in Section 4. 



2 Model 



Let us consider a well-mixed population of constant and finite N individuals 
. Suppose the strategy set in our model is A, B and C. The payoff matrix of 
the three strategies is 
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The fitness of individuals using A, B and C is respectively as follows: 

= l-w + w[a{z - 1) + bj +p{N-i- j)]/{N - 1) 
gij ^l-w + w[ci + d(j -l) + q(N-i- j)]/(N - 1) (3) 
hij ^l-w + w[mi + nj + l(N -i-j - 1)]/(N - 1) 
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Here i denotes the number of individuals using strategy A, j denotes the 
number of those using strategy B, and there are N — i — j players using 
strategy C. The balance between selection and drift can be described by a 
frequency-dependent Moran process. At each time step, the number of A indi- 
viduals increases by one corresponding to two situations. One is eliminating a 
B individual whereas the number of C players keeps unchanged. The other is 
eliminating a C individual whereas the number of B players keeps unchanged. 
The transition probabilities can be formulated as: 



ifi,j + 39i,3 + {N -i- j)hij TV 
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Here Pif+i is the transition probability from the state of i A, j B and N — i—j 
C individuals to that ol i -\- 1 A, j B and N — i— j — IC individuals. Let 
x{i,j) denotes the fixation probability that i A individuals could invade the 
population of j B and N — i—j C individuals. We have the recursive relation: 

x{i,j) = x{i + l,j)Pi:U + x{i + l,j- 

+ x{i - i,j)Pi;U + x{i - i,j + i)P^i^ (5) 

+ x{ij + l)P^f+' + x{ij- V)Pif' + 



Researchers reported that in a well-mixed environment, two of the initial three 
kinds of strategists would go extinct after some finite time, while coexistence 
of the populations was never observed [25]. In finite populations, no matter 
how many kinds of individuals initially, only one type of strategists can survive 
in the evolutionary game eventually. Hence, the fixation probabilities of the 
game with two strategies can be viewed as special boundary conditions of our 
model. There are three types of boundary conditions: 

(1) obviously, .t(0, j) = 0, j = 0, 1, ■ • • , A^; 

(2) a;(z, 0) = x{i)^ i = 0, 1, • ■ • , A^, here x{i) is the fixation probability that i A 
individuals could take over the population of iV — i C and no B players; 

(3) similarly, x{i, N — i) — x{i), i — 0,1, • • ■ , N, here x{i) means the fixation 
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probability that i A individuals could invade the population oi N — i B and 

no C players. 

Note that x{i) can be formulated as Eq. 2. Similarly, x{i) is written as 



j=l fc=i Jk 
j=l k=l -Ik 



xn] 



here i = 1, 2, ■ • ■ , iV - 1, = 1 -w + w[a{i - 1) + p{N - i)]/{N - 1), 
hi = 1 — w + w[mi + 1{N — i — 1)]/ (A^ — 1), and the corresponding boundary 
conditions are x{0) — 0, x{N) ~ 1. 

The relationship among the solutions of the system of equations can be de- 
picted by Fig. 1. The point (i, j) marked by full black dot denotes the boundary 
condition of the equations, while this marked by empty dot denotes the un- 
known of the equations. Thus in what follows, the unknown element x{i,j) of 
our interest is discussed. 




1 2 3 4 5 N 
# of A individuals 



Fig. 1. The point marked by full black dot means the boundary condition of 
the equations, while this marked by empty dot means the unknown of the equations 
which could be formulated by the boundary conditions. 

Eq. 5 can be transformed to Eq. 6 which is a system of linear equations in 
{N -1){N - 2)/2 variables. 



[i{N - i)fi^, + j{N - j)g,j + {i+j){N-z- j)hij]x{i,j) 
^ i{N -i- j)hijx{i - 1, j) + ijgi,jx{i - IJ + 1) 
+ i{N -i- j)fi,jx{i + + ijfi,jx{i + - 1) 
+ j{N - i - j)gijx{i,j + 1) + j{N - i - j)hijx{i,j - 1) 



(6) 



Where i = 1,2, ■ ■ ■ , N — 2, j = 1,2, ■ ■ ■ , N — i — 1. Accordingly, Eq. 5 can 

— * 

also be simplified to Ax — b, where x is a vector (a;(l, 1), x{l, 2), • • • , x{l, N — 
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2), ,t(2, 1), ,x(2, 2), • • ■ , x{2, N-3), x{i, 1), x{i, 2), • ■ • , x{i, N-i-1), x{N- 
2,l)f,Ais the corresponding coefficient (iV - 1) (iV - 2)/2 x (A^ - 1) (A^ - 2)/2 
dimensional matrix, b is the corresponding vector composed of these boundary 
conditions. Matrix A can be written as follows: 



An-2,N-2 An-2,N-3 

^Ar-3,iV-2 ^iV-3,Ar-3 ^iV-3,Ar-4 

Ai\i_4^N-3 ^Ar-4,7V-4 ^Ar-4,iV-5 
























'■N-5,N-4 ^N-5,N-5 ^N-5,N-6 ' ' 



\ 



^2,3 ^2,2 ^2,1 
^1,2 ^1,1 ) 



(7) 



Here A]\f_i^j^_j states a. {N — i) x (A^ — j) block of matrix A. 
For fixed i, in the sub-vector {x{i, l),x{i, 2), • • • , x{i, N — i — 1))^, there exist 
a maximal probability Xmaxii) and a minimal probability Xmin{i)- They both 
play significant roles in the evolution process. Note that in evolutionary dy- 
namics, we have Xmaxi'^) < Xmax{2) < ■ ■ ■ < Xmax{N-2), < Xrnin{2) < 

■ ■ ■ < Xmin{N — 2). If a single A individual can be favored to replace B or C, 
more than one A individuals are more likely to replace B or C. Thus we only 
focus on the fixation probabilities x{l,j). For a neutral mutant, the fixation 
probability is When x{l,j) > 1/N, natural selection favors A replacing 



B or C. We define Xr, 



as ^f/o&a/ fixation probability and Xmaxii) as local 



fixation probability oi i A individuals. If x„tira(l) > 1/^- natural selection 
favors A replacing i? or C no matter what the ratio of 5 to C is , that is 
global. Similarly, if Xmaxi^) > ^/N, the selection favors A replacing S or C in 
some proper ratios of B to C, not any ratios, that is local. Accordingly, there 
may be three situations: 

(1) if Xmaxi^) < 1/-^; natural selection never favors A replacing B and C; 

(2) if Xjnin{l) > i/N, natural selection always favors A replacing 5 or C 
whatever the ratio of S to C is. It is likely that A could invade the population; 



(3) if a;^i„(l) < 1/N and x^axi^) > ^/N, natural selection favors A replacing 
S or C in some proper ratios of B to C. Thus the fixation of A is possible 
under some suitable conditions. 



Let us compare the evolutionary game dynamics in infinite populations with 
that in finite populations. The finite size effects bring stochastic factor to the 
evolution. A individuals which are eliminated in infinite populations may be 
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favored by natural selection replacing B or C individuals. In Fig. 2, the left col- 
umn shows evolutionary game with a small frequency of A individuals initially 
in infinite populations, and the right column shows that with a single A indi- 
vidual at first in finite populations. The first row shows situations with payoff 
matrix a — 2, b — A, p — A, c — 3, d — 5, q — 1,171 — 3, n — 1,1 — 5, the size 
N — 50, w = 0.1. A individuals will disappear in infinite populations no matter 
how many B and C individuals are initially. While in finite populations, the 
selection won't favor A replacing B or C. The second row shows the situations 
with payoff matrix a — 5,b — 4,p = A,c = A,d = 3,q = A,m = 4,n = 4,l = 3, 
the size N — 50, w — 0.1. A individuals will always invade the population 
of B and C individuals in infinite populations. And in finite populations, 
natural selection will all the time favor A replacing 5 or C no matter what 
the ratio of 5 to C is. The third row shows situations with payoff matrix 
a — 5,b — 5,p — 1.6, c = 5,d = 5,q = 0,171 = 3,n = 7,1 = 2, N = 50,w — 0.1. 
A individuals will disappear in infinite populations no matter how many B 
and C individuals are initially. Furthermore, from plenty of computations, we 
find that A individuals will monotonously decrease to zero in infinite situation. 
Whereas sometimes A will tend to replace or C in finite populations. 



For the three payoff matrixes in Fig. 2, it is clear that the fixation probability 
of A individual is not always monotonic function of the number of B indi- 
viduals (see Fig. 3). Thus, the global and local fixation probability of A may 
nontrivially occur at intermediate ratio of B to C, not always end points of 
the ratio range of B to C. 



3 AllC-AUD-TFT 



Let us consider a very interesting and famous repeated game with three strate- 
gies AUG (cooperate all the time), AUD (defect all the time) and TFT (tit- 
for-tat). TFT is an adaptive cooperative strategy which is one of the most 
successful strategies proved by experiments. Individuals using TFT cooperate 
in the first round generally, and then do whatever the opponents did in the 
previous round. The number of rounds n, by definition, can be 1, 2, • • ■ , oo. 
If the rounds are infinite, this game is out of our consideration. In one-round 
repeated Prisoners' Dilemma game, a cooperator can obtain a benefit of b or 
if it meets a cooperator or defector, meanwhile, it must cost c whomever its 
opponent is; a defector can obtain a benefit of 6 or if it meets a cooperator 
or defector, but costs nothing in the whole process. We bring the ratio of cost 
to benefit, by definition, r (r = c/6g[0,1]), into our game, the payoff matrix 
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Fig. 2. The comparisons between different situations corresponding to infinite and 
finite populations respectively. In the left column figures, the x— axis denotes the 
percentage of B individuals initially, meanwhile, the y— axis denotes the percentage 
of A individuals after 1000 steps. In the right column figures, the x— axis shows the 
size of the population, and the y— axis shows the fixation probability of a single A. 
Let NFP, LFP and GFP represent the neutral fixation probability the local 

and the global fixation probability respectively. 

between cooperator and defector can be simplified as follows: 

C D 
C 1 
D 1 + r r 
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Fig. 3. The fixation probability of only one A individual as a function of the number 
of B individuals. Where = 50, NFP is the neutral fixation probability, PMz means 
the situation with payoff matrix i in Fig. 2, i = 1,2,3. The fixation probability 
of only one A individual is not always monotonic function of the number of B 
individuals. The global and local fixation probability may be not at the end points 
of the curve. 

Thus the payoff matrix of TFT, AUG and AUD with n rounds is 

TFT AllC AUD 
TFT n n {n — l)r 

AUC n n 

AUD 1 + nr n{l + r) nr 



The pairwise comparison of the three strategies leads to the following conclu- 
sions. 

(1) AllC is dominated by AllD, which means it is best to play AllD against 
both AllC and AllD; 

(2) TFT is equal to AllC when TFT plays with AllC; 

(3) If the average number of rounds exceeds a minimum value, n > 1/(1 — r), 
then TFT and AllD are bistable. 

Suppose that a single individual using strategy TFT is brought in the popula- 
tion in which some individuals adopt strategy AllC and the others use strategy 
AllD originally. Provided that the number of rounds is finite and greater than 
1/(1 — r), the strategies TFT and AllD are both strict Nash equilibrium and 
evolutionary stable strategies (ESS) [2]. If n = 1, TFT becomes strategy C 
which is out of our discussion. Thus we need finite n ^ 2. For a fixed number 
of individuals and a value of the rounds, we find that there is a barrier of r 
which can determine whether or not the selection favors TFT replacing AUC 
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or AUD. The barrier also has two types: one is ri, which represents the barrier 
of local situation; the other is Vg, which represents that of global situation. 
If r < ri, the natural selection favors TFT replacing AUG or AllD locally, 
whereas if r > r^, TFT tends to be washed out by selection. The results about 
Tg are similar. Under the condition of weak selection, for sufficiently large 
population size and large number of rounds n, the barrier ratio r as the 
function of n is approximately followed by {n — l)/{n + 6), here 6' is a param- 
eter dependent with A^. In Fig. 4, we can fit ri and Vg as r; = (n — 1)/(?t, + 2), 
and Tg = [n — l)/{n + 2.6). In general, for w <C 1, sufficiently large A^ and 
large n, from abundant numeric computations, we obtain r; = {n — l)/{n + 2) 
and Tg = {n — l)/{n + 6g), here 6g is also a parameter dependent with A^ and 
6g > 2. Thereforce, r; > Vg. In other words, the ratio r which could lead the 
selection to favor TFT replacing AUG or AllD globally can also induce local 
replacement, but not vice versa. 

Deterministic replicator dynamics with three strategies in infinite populations 




Fig. 4. The global and local barrier ratios rg and r; as a function of the number of 
rounds n. Here N = 50, w = 0.1. r'g represents the fitting curve of Vg, r[ represents 
the fitting curve of r^. We can fit ri and rg as = and rg = ^^2% - 

admits two interior equilibria at frequency of TFT given by x* = r/(n — 1)(1 — 
r) and x** = nr/{n — 1). For local situation, substitude r < {n — l)/{n + 2) 
into X* and x**, we get x* < 1/3 and x** < n/{n + 2); for global situation, 
substitude r < {n - l)/{n + 9g) into x* and x** , we obtain x* < 1/(1 + 9g) 
and X** < n/{n + 6g). If the frequency of TFT at the equilibrium is x* < 1/3 
or X** < n/{n + 2) in infinite populations, it will be favored replacing AUG or 
AllD locally in finite situation; if the frequency of TFT at the equilibrium of 
infinite situation is x* < l/{l + 6g) or x** < n/{n + 6g), it will tend to replace 
AUG or AllD globally by TFT in finite populations. 

The 1/3 law proposed by Nowak in [12] is still valid in our case, that is, the 
selection favors TFT replacing AUG or AllD in finite populations, if its fre- 
quency at the equilibrium is x* < 1/3 in infinite populations. However, when 
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there are three strategies TFT, AUG and AUD, in which TFT's frequency 
at one equihbrium is x*. the corresponding frequency of AUG is zero. Thus 
in this situation, our results to some extent vahdate the conjecture in which 
AUG is ehminated by natural selection so quickly that the effect of AUG can 
be neglected in finite populations. And then the evolutionary game dynamics 
with the two left strategics TFT and AUD, is equivalent to the situation of 
the situation with these two strategies initially. Nevertheless not any size of 
AUG individuals could be wiped out quickly, their effects can not be ignored 
in the dynamics. Hence x* < 1/3 can only determine the replacement locally 
(in some certain circumstances). As for global fixation situation (the fixation 
is certain for any ratio of AUG to AUD), we have x* < + 6g) < 1/3 for 
6g > 2. The conditions that natural selection favors g'/o&a/ replacement of AUG 
or AUD by TFT are more intensified than those of local situation. 
Let us discuss the other equilibrium. The n/ {n + 6*) is a monotony increasing 
function of n. This n/ {n + 6) approaches one for increasing n. That is to say, 
when the number of rounds increases, the condition x** < n/{n + 9) can be 
satisfied with higher probability, and TFT may have more opportunities to 
replace AUG or AUD locally and globally. In the standard evolutionary model 
of the finitely repeated Prisoner's Dilemma, TFT can not invade AUD. But 
interestingly, we find that for intermediate n, if x** < n/{n + 9), nature selec- 
tion favors TFT replacing AUG or AUD as the fixation probability of a single 
TFT (ptft) is larger than that of a single AUG (pauc) ot AUD (paud)- Ac- 
tually, in this case, Ptft > 1/A^ > 'n^O'X{pAUD ^ Pauc) ■ Therefore, a single TFT 
is likely to invade the entire population consisting of AUG and AUD finally. 
In this case, cooperation tends to emerge in the evolution process. And yet, 
for large limit n, the situation is out of our consideration due to its extraor- 
dinary intricacy. However, as n increasing to infinite, the probability that the 
selection favors TFT taking over the whole population also approaches one. 
Accordingly, the fixation of cooperation is enhanced in finite populations. It is 
because that when TFT meets AUD, its loss in the first round can be diluted 
by many rounds games. In this case, the total fitness of TFT is almost the 
same as that of AllD and they are a pair of nip and tuck opponents. But TFT 
receives more payoff than AllD when they both play with TFT. To sum up, 
TFT is superior to AllD for limit large rounds because of its adaption. As 
a result of this property of TFT, natural selection mostly prefers to choose 
TFT to reproduce offspring, and then TFT is most Ukely to dominate the 
population at last. Therefore, cooperation has more opportunities to win in 
finite populations contrasting against infinite situation. 
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4 Conclusion 

We have proposed a model of evolutionary game dynamics with three strate- 
gies in finite populations. It can be characterized by a frequency-dependent 
Moran process which could be stated by a system of hncar equations. By the 
comparative study of evolution in finite and infinite populations, we shew that 
a single A individual which can not invade infinite populations may have an 
opportunity to replace S or C in finite situation. In other words, a single 
A individual could be eliminated by selection with smaller probability in fi- 
nite populations than situation in infinite populations. In addition, a famous 
game with AllC, AllD, and TFT is adopted to illuminate our results by nu- 
meric computations. Furthermore, under the condition of weak selection, for 
sufficiently large population size N and appropriate number of rounds n, a 
single TFT could invade the population composed of AllC and AllD with high 
probability almost one. In this situation, the emergence of cooperation is at- 
tributed to the finite population size effects. Our results may help understand 
the coevolution of multi-species and diversity of natural world. 
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